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Subject: The Department of Homeland Security Needs to Fully Adopt a 

Knowledge-based Approach to Its Counter-MANPADS Development 
Program 

In late 2002, terrorists fired surface-to-air missiles at an Israeli airliner departing from 
Mombasa, Kenya — the first time man-portable air defense systems (MANPADS) had 
been used to attack commercial aircraft in a non-combat zone. Given concerns about 
the vulnerability of the commercial airline industry and the potential impact of an 
attack in the United States, you requested that we conduct an assessment of the 
federal government's efforts to address the MANPADS threat against commercial 
aircraft, including its nature and extent; the Department of Defense's monitoring of 
Stinger missiles exported to other countries; and U.S. bilateral and multilateral efforts 
to address international MANPADS proliferation. After we began work on this 
assessment, the Department of Homeland Security (DHS) took steps to initiate a 
2-year system development and demonstration program for a counter-MANPADS 
system and awarded the initial contracts in January 2004. On December 4, 2003, we 
briefed your staff on our views about DHS's approach to developing the system. This 
report summarizes that information and transmits the portion of the briefing related 
to DHS's counter-MANPADS development effort (see encl. I). Our assessment of the 
other federal efforts to address the threat is ongoing, and we expect to complete our 
report in the spring of 2004. 
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Background 

Due in part to the Mombasa attack, the White House convened a task force to 
develop a strategy to reduce the MANPADS threat against commercial aircraft. In 
reviewing available technical countermeasures, the task force identified an on-board 
jammer (directed infrared countermeasure, or DIRCM) as the most promising 
technology to meet current threats while potentially satisfying operational 
constraints imposed by the commercial aircraft industry such as minimizing the cost 
to operate and maintain these systems. 

The Congress directed DHS to submit a plan to develop and demonstrate a 
counter-MANPADS device for commercial aircraft. 1 On October 3, 2003, DHS 
released a solicitation that outlines a 2-year, two-phased system development 
and demonstration program to produce prototype systems that would satisfy 
performance, operational, and cost constraints. In Phase I, which begins in 
January 2004, DHS intends to conduct preliminary design and analysis activities. 
In Phase II, which begins about 6 months later, they plan to develop and test the 
prototypes. 

The objective of the DHS program is to (1) migrate existing military countermeasure 
technologies to the civil aviation environment and (2) minimize the total life-cycle 
cost of the system, which includes development, procurement, installation, operation 
and support costs. The solicitation focuses primarily on the DIRCM concept, which 
combines a missile warning system (MWS) to detect a missile launch and a laser to 
jam the guidance system of the missile. DOD currently uses DIRCM technology on 
some of its large transport aircraft, such as the C-17. 

Results in Brief 

DHS faces significant challenges in adapting a military counter-MANPADS system to 
commercial aircraft. These challenges include establishing system requirements, 
maturing technology and design, and setting reliable cost estimates. For instance, 
DHS has to account for a wide variety of aircraft types in designing and integrating 
the system. Our past work on the best practices of product developers in government 
and industry has found that the use of a knowledge-based approach is a key factor in 
successfully addressing such challenges. This approach includes the use of exit 
criteria or controls to ensure that sufficient knowledge has been attained at critical 
phases of the product development process. Based on input we provided during the 
course of our review, DHS updated its initial solicitation to incorporate these 
knowledge-based exit criteria. We think this a positive first step, and we are 
recommending that the Secretary of Homeland Security ensure that the knowledge- 
based approach is fully implemented throughout the course of its counter-MANPADS 
development program. DHS fully concurred. 



1 House Report 108-76, p. 84. 
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DHS Faces Challenges in Adapting Military Counter-MANPADS System to 
Commercial Aircraft 

In proceeding through Phases I and II of the counter-MANPADS development 
program, DHS intends to establish system requirements, mature technology and 
design, and set reliable cost estimates. Such issues are interrelated and their 
resolution will have a direct impact on DHS's ability to effectively implement its 
program. A brief discussion of these program issues follows: 

• Requirements involving new technologies, system maintenance, system 
integration, and system security for the counter-MANPADS system are to be 
developed and this may involve trade-offs between competing objectives. For 
example, DHS intends to study the trade-off of system performance objectives 
with total cost to derive the most effective solution at a realistic life-cycle cost. 

• Technology and design problems include a high false alarm rate affecting the 
current generation of Missile Warning Systems (MWS) used by the military; 
changes needed to adapt military countermeasures to commercial aircraft; and 
the use of classified jam codes by civilian aircrews. MWS's current high false 
alarm rate may increase system failures. Whenever the MWS detects a missile 
launch, the system cycles. If a false alarm has been received and the system is 
cycling unnecessarily, it will reach its mean-time-between-failure (MTBF) rate 
threshold much faster. A new MWS that would resolve this issue is being 
developed but is not yet mature. The solicitation also requires a "call back 
notification system" that would alert air traffic controllers once it detects a 
missile launch, but it has not been developed. A MWS with a high false alarm 
rate connected to a call back system could cause unnecessary airport closures. 

The DHS solicitation also requires a system reliability MTBF rate of greater 
than 3,000 hours — 10 times the current rate for DIRCM systems on military 
aircraft of 300 hours. A low reliability rate drives the operations and support 
costs and the ability of the airlines to maintain the system. DHS also intends to 
require a common system attached to the wide variety of commercial aircraft, 
even though the design, placement, and integration of this system are all 
unknowns that will affect each aircraft type differently. For example, a system 
container that causes a 1-percent excess drag on a Boeing 747 aircraft will 
cause greater drag on a smaller 737-model aircraft, which will affect the 
airlines' fuel consumption and increase costs. 

Design issues surround the classified jam codes used in DIRCM. Military pilots 
are cleared to handle the classified material and military aircraft are 
safeguarded, but DHS has not yet developed a plan to handle the classified 
material and safeguard the hardware on a commercial aircraft. According to 
DHS officials, any system developed will need to have a tamper-proof design. 

• Operational test requirements are not clearly specified by the solicitation. 
DHS compressed its development cycle to comply with congressional 
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direction. According to program officials, DHS has acted to reduce the time 
and cost required to conduct testing by having the contractor conduct required 
testing. However, DHS intends to approve both test concepts and the test 
plans. They stated that DHS will closely monitor all contractor conducted 
testing to ensure tests are correctly executed and accurately reported. Finally, 
operational testing will be conducted on DOD ranges using its test facilities. 
DOD uses independent live fire and operational test and evaluation that are 
not under the control of the developers to demonstrate that the developed 
system is suitable and effective before authorizing full-scale production. 

• Reliable cost estimates regarding the procurement, integration, operation, 
and support of DIRCM system on commercial aircraft do not currently exist. 
DHS intends to conduct its own cost estimates and also require independent 
analysis, which will be difficult because (1) DHS does not know how many 
units will be required and, therefore, cannot determine a price based on 
economies of scale; (2) costs for integrating the system on different types of 
aircraft are not yet determined; and (3) the reliability rate is unknown and 
therefore operations and support costs are difficult to estimate. 

In addition, the production of a large number of countermeasure systems 
quickly would probably require a significant capital investment to increase 
production capacity. An industry official placed their current production 
capability at roughly four DIRCM systems per month. According to a program 
official, DHS does not currently know how many or what combination of the 
estimated 6,800 aircraft in the U.S. fleet might be equipped. One option would 
be to initially equip the planes of the Civil Reserve Air Fleet (CRAF), 2 but doing 
even that could take years. 

Knowledge-based Approach Has Been Adopted by Successful Product 
Developers 

In the last several years, we have undertaken a body of work on how leading 
developers in industry and government use a knowledge-based approach to develop 
products that reduces risks and increases the likelihood of successful outcomes. This 
best practices model enables decision makers to be reasonably certain about their 
products at three critical junctures or knowledge points during development and 
helps them make informed investment decisions. 

• Knowledge Point 1 : Before product development is started, a match must be 
made between the customer's needs and the available resources — which 
include technical and engineering knowledge, time, and funding. 



2 The CRAF Program enlists the U.S. airline industry to help create an increased airlift capability for 
contingencies. Airlines sign contracts with the government entitling Air Mobility Command (AMC) to 
mobilize the aircraft and their aircrews when airlift needs exceed the capability of military aircraft. 
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• Knowledge Point 2: A product's design must be able to meet performance 
requirements and become stable about midway through development. 

• Knowledge Point 3: The developer must show that the product can be 
manufactured within cost, schedule, and quality targets and is demonstrated 
to be reliable before production begins. 

A knowledge-based approach also involves the use of controls or exit criteria to 
ensure that the required knowledge has been attained at each critical juncture. It 
ensures that managers will (1) conduct activities to capture relevant product 
development knowledge, (2) provide evidence that knowledge was captured, and 
(3) hold decision reviews to determine that appropriate knowledge was captured to 
allow a move to the next phase. If the knowledge attained at each juncture does not 
confirm the business case on which the effort was originally justified, the program 
does not go forward. 

Use of a knowledge-based approach has enabled leading organizations to deliver high 
quality products on time and within budget. Product development efforts that have 
not followed a knowledge-based approach can be frequently characterized by poor 
cost, schedule, and performance outcomes. 

DHS Has Begun to Use Knowledge-based Approach 

DHS included a number of knowledge-based elements in its original solicitation 
released on October 3, 2003. Among other things, it plans to conduct design reviews, 
require periodic performance assessments from the contractor, utilize an integrated 
product team to identify and resolve issues, conduct systems engineering work in 
both phases, and require the development of a prototype to help identify and resolve 
specific design and manufacturing risks. In terms of systems engineering, for 
example, DHS intends to use a comprehensive, iterative technical management 
process that includes translating operational requirements into configured systems, 
integrating technical inputs, managing interfaces, and characterizing and managing 
risks. 

During our review, we asked DHS to identify its controls or exit criteria for use in 
determining whether needed knowledge had been attained by the end of Phases I 
and II of its program. In its solicitation of October 3, 2003, DHS had required the 
contractor to satisfy certain criteria in order to receive payment for each milestone. 
However, the Phase I exit criteria were not knowledge-based. Rather, they were 
based on the contractor providing information, such as the Long Lead Items List, at 
key payment milestones. They did not require the contractor to demonstrate that key 
product knowledge has been obtained. Also, the Phase II exit criteria were not 
identified and were to be proposed by the contractor and subject to negotiation. 

We presented DHS officials with recommended exit criteria from our past reports 
(see examples in encl. I, p. 20), and they agreed to integrate them into an updated 
solicitation and use them in monitoring the contractors' progress. For example, at 
Knowledge Point 1, exit criteria include the demonstration that critical technologies 
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are mature and system requirements are finalized. At Knowledge Point 2, criteria 
include the completion of 90 percent of engineering drawings at design review and 
the demonstration that a prototype's design meets requirements. At Knowledge 
Point 3, criteria include the demonstration that manufacturing processes are under 
statistical control. 

Conclusion 

To address the significant challenges involved in adapting a military counter- 
MANPADS system to commercial aircraft, DHS would benefit from fully adopting the 
knowledge-based approach used by leading developers in government and industry. 
This approach is predicated on the use of exit criteria at each phase of the 
development process to ensure that needed knowledge is attained before proceeding 
to the next phase. To their credit, DHS officials responsible for this effort have 
agreed to this approach, successfully incorporated exit criteria into their updated 
solicitation, and agreed to use them to monitor progress. 3 This is a positive first step, 
but DHS needs to ensure that the knowledge-based approach is fully utilized 
throughout this development effort. 

Recommendation for Executive Action 

We recommend that the Secretary of Homeland Security fully adopt the knowledge- 
based approach, including the use of exit criteria, to help ensure that key decisions in 
DHS's effort to develop and demonstrate a counter-MANPADS system are based on 
sufficient information. 

Agency Comments and Our Evaluation 

In written comments to a draft of this report (see encl. II), DHS fully concurred with 
our findings and recommendation. DHS also provided separate technical comments, 
which we have incorporated as appropriate. 

Scope and Methodology 

In conducting our review, we compared DHS's plan for its counter-MANPADS system 
development and demonstration program plan against the best practices of 
commercial and military acquisitions identified in our past reports and focused 
whether DHS will have sufficient information to make knowledge-based decisions at 
each milestone. To determine what military countermeasures are available for 
adaptation to commercial aircraft and what their performance capabilities, cost, and 
schedule ramifications might be, we met with DOD, Air Force, Army, and Navy 
officials and analyzed relevant documentation, including studies and test reports. 
We interviewed representatives from Northrop Grumman, Boeing, BAE Systems, 
Raytheon Corporation, and Sanders Design International regarding countermeasure 
systems currently in production or development. We also met with representatives 



Examples are included on page 20 of the enclosed briefing. 
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from the airline industry, air transportation association, and RAND. We conducted 
our work from April 2003 through January 2004 in accordance with generally 
accepted government auditing standards. 



We are sending copies of this report to the Secretary, Department of Homeland 
Security; the Director, Office of Management and Budget; and other interested 
congressional committees. Copies are available to others upon request. In addition, 
the report will be available at no charge on the GAO Web site at http://www.gao.gov. 

Should you or your staff have any questions on matters discussed in this report, 
please contact me on (202) 512-4841 or Jim Morrison at (202) 512-7078. Principal 
contributors to this report were Mike Aiken, Natalie Britton, Terry Parker, and 
Richard Strittmatter. 




Robert E. Levin 

Director, Acquisition and Sourcing Management 



Enclosures 
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DHS's Exit Criteria Are Not Linked to Knowledge 



Briefing to the House Committee on 
Armed Services and House Committee 
on Transportation and Infrastructure 
Subcommittee on Aviation 
December 4, 2003 
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MANPADS Threat: Commercial Aircraft 
Are at Risk 



• Easily transportable and concealable, 
and relatively inexpensive (less than 
$1,000 to $100,000) 

• Some training required to use system 
effectively 

• 35 MANPADS attacks on commercial 
aircraft since 1 978 

• All but one (November 2002) 
occurred in combat zones 




U.S. Stinger missile system 
Source: CRS / U.S. Army 
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MANPADS Threat: Systems Are Widely 
Available 



• 20 countries manufacture MANPADS; nearly all countries possess them 

• Estimated 500,000 - 750,000 worldwide, both in national arsenals and on the 
black market 

• Many different missile types and capabilities; new versions under development 

• MANPADS from Iraq present new security challenges 
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MANPADS Threat: Systems Are Widely 
Available 

• Determining actual threat is difficult 

• Numbers on black and gray markets difficult to estimate 

• Effective lifetime of systems uncertain 

• U.S. has not performed assessment of where MANPADS threat falls 
relative to other terrorist threats 

• Officials acknowledge that consequences of a MANPADS attack on 
a U.S. commercial aircraft would be severe 

• Difficult, therefore, to prioritize and allocate resources to the many 
potential terrorist threats facing the U.S. 
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White House Task Force Recommendation 
and DHS Response 



Task Force Recommendation 

• Identified on-board jammer (directed infrared countermeasure-DIRCM) 
as most promising existing technical countermeasure because of 
limitations with other systems 

• Ground-based systems provided limited coverage 

• Expendables posed environmental hazard 

• Lamps offered limited capability 

DHS Actions 

• Submitted congressionally mandated plan to Congress (May 22, 2003) 
outlining development and demonstration of an anti-missile device for 
commercial aircraft 



• Released counter-MANPADS development and demonstration solicitation 
(Oct. 3, 2003) 
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DHS Program Plan 

Primary Objectives 

• Migrate current countermeasure technologies from the military to the civil 
aviation environment 

• Focus is on DIRCM, but other alternatives will be considered for the first 
phase of the program 

• DIRCM is effective against current threats 

• Minimize total life-cycle cost of system 

• Willing to trade requirements to minimize life-cycle cost 

Ambitious 24-Month Program Schedule 

• Two-phase Systems Development & Demonstration (SD&D) beginning Jan. 2004 

• Phase l-Preliminary Design & Analysis for two or more systems (6 months) 

• Phase ll-Prototype Development & Testing for up to two systems (18 months) 
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Knowledge-Based Approach Can Lead to 
Better Acquisition Outcomes 



• GAO's past reviews of commercial and defense practices in product 
development have identified best practices 

• Successful programs reduce risk by ensuring that high levels of knowledge 
are achieved at key points in development 

• Using exit criteria linked to knowledge will ensure that program has 
sufficient information to proceed 

Knowledge Point 1 

Achieve match between a customer's needs and the developer's available 
resources — technology, time, engineering knowledge, and funding — before 
program start 

Knowledge Point 2 

Stabilize design midway through development and meet performance 
requirements before initial manufacturing begins 

Knowledge Point 3 

Determine that product meets cost, schedule, and quality targets and 
demonstrate reliability before production begins 
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Ambitious Program Schedule Presents 
Challenges for DHS 



Knowledge-Based Approach 

Mission needs Requirements 



understood 



Technologies 
mature 



Technology 
Development 



set 
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Design 
stable 
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Product can be produced 
within cost, schedule, 
and quality targets 



Product Development 
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DHS Program Plan 
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Production 
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Technology 
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Product Development 
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Production 
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Integration Demonstration 





DHS Start KP 1 ? 
Jan. 04 



KP 2 ? 



Production 
Decision 
KP 3 ? 
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DHS Program Plan Incorporates Many Aspects 
of a Knowledge-Based Acquisition Process... 



• Preliminary Design Review (PDR) -evaluates progress, technical adequacy, 
proposed software architectures and risk resolution of selected design approach 

• Critical Design Review (CDR) -determines that design satisfies performance 
and engineering requirements 

• Program Management Reviews -requires periodic performance assessments 
from contractor 

• Integrated Product Team -includes representatives from appropriate disciplines 
working together to build successful programs, identify and resolve issues, and 
make recommendations 

• Systems Engineering -uses comprehensive, iterative technical management 
process that includes translating operational requirements into configured 
systems, integrating technical inputs, managing interfaces, and characterizing 
and managing risk 

• Prototype Development -assists in identification and resolution of specific 
design or manufacturing risks 
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But Areas of Concern Remain 



• Mission Need 

• United States intelligence community has not completed a comprehensive 
assessment of the MANPADS threat in the context of other terrorist threats 

• U.S. faces a variety of threats within its borders 

• Likelihood and impact of MANPADS attack compared to other terrorist 
threats is unknown 

• Mission needs will not be fully articulated at program start 

• Alternatives 

• DHS has not fully analyzed alternatives to an on-board jammer 

• The level of effort that non-DIRCM options will receive is unclear 

• Requirements 

• Will not be clearly or completely defined at program start 
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But Areas of Concern Remain (cont.) 



• Technology & Design 

• Some critical technologies are immature and require development 

• Laser-unclear if miniaturization of the laser turret will meet 
required reliability rate (Mean Time between Failure - MTBF) 

• Some design specifications require new technology 

• Missile warning system (MWS) 

• Call-back notification 

• Design, placement, and integration of system container are unknowns 

• Plan for handling and safeguarding classified hardware and software 
unknown 

• Requires tamper-proof design 
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But Areas of Concern Remain (cont.) 



• Production 

• Production capacity is currently unknown 

• Prototype may not be production representative 

• Testing 

• Heavy reliance on contractor testing 

• Testing required during Phase I limited to modeling and simulation 

• Reliability will not be adequately demonstrated in Phase II 

• Cost assessments 

• Cost analysis scheduled during Phase I will be assumption-based 

• Unit costs unclear; number of units required unknown 

• Integration costs unknown 

• O&S costs unclear; i.e., MTBF unknown 

• Potential unit and O&S costs of equipping and maintaining countermeasures 
systems on aircraft are high 

• Cost and scope of potential contract is unknown 
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Linking Best Practice Exit Criteria to Knowledge 
Will Help Reduce Risk 

KP I Exit Criteria KP 2 Exit Criteria KP 3 Exit Criteria 



• Demonstrate maturity 
of critical technologies 

• Complete trade-offs 
and finalize 
requirements 

• Complete initial cost 
and schedule 
estimates using 
results from 
preliminary design 
review 



Decision review for 
program launch 



• Complete 90% of 
engineering drawings by 
CDR 

• Demonstrate with 
prototype that design 
meets requirements 

• Complete failure modes 
and effects analysis 

• Identify critical 
manufacturing processes 

• Identify reliability rate 
targets and growth plan 



Decision review to enter 
system demonstration 



• Demonstrate 
manufacturing processes 

• Build and test production 
representative prototypes 

• Demonstrate in 
operational environment 

• Demonstrate critical 
processes are capable 
and in statistical control 



Decision review to 
begin production 
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DHS's Exit Criteria Are Not Linked to Knowledge 



• DHS will use exit criteria linked to progress payments at selected milestones 
to monitor and reward contractor performance 

• Phase I exit criteria are not knowledge-driven 

• Phase II exit criteria have not been identified. These criteria 

will be proposed by the contractor and are subject to negotiation 
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Conclusions 



• DHS's plan includes many elements of a knowledge-based 
approach to product development, including 

• PDR and CDR • Program management reviews 

• Systems engineering • Integrated product team 

• Prototype development 

• Despite this, they will lack sufficient information to: 

• Develop accurate cost estimates 

• Know if product will work as intended 

• Understand if product can be manufactured as designed 

• Until comprehensive threat assessment is completed, mission need will not be 
fully understood 

• Current exit criteria are event-based, not knowledge-based 
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Recommendations 



• DHS needs to fully adopt a knowledge-based approach in its acquisition 
process and implement the type of exit criteria outlined in this briefing to ensure 
that each decision is based on sufficient information. 
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Comments from the Department of Homeland Security 



U.S. Department of Homeland Security 

Washington, DC 20528 

|H Homeland 
Security 

January 22, 2004 
R.E. Levin 

Director, Acquisition and Sourcing Management 
United States General Accounting Office 

Dear Mr. Levin: 

Thank you for the opportunity to comment on your GAO Draft Report, dated January 12, 2004, 
titled: The Department of Homeland Security Needs to Fully Adopt a Knowledge-based Approach to 
Its Counter-MANPADS Development Program. As you know, the Department of Homeland 
Security considers the Counter-MANPADS program one of its vital initiatives in air transportation 
security for the American people. We appreciate the GAO's participation effort, and your insights 
and suggestions in the execution of the program. As we launch into the validation phase of 
transitioning existing military technologies to civilian aircraft, there are many technical, schedule, 
and cost unknowns. 

We fully concur with your assessment that knowledge-based evaluations of technical requirements, 
schedules, total system cost estimates, etc. are essential to the success of this program. In fact, each 
of the contractors selected to perform phase one of this effort have been told the Government 
evaluators are employing knowledge-based evaluations throughout the program. Based on the 
GAO's preliminary recommendations, we had developed preliminary milestone exit criteria for the 
solicitation. 

For your consideration we have included two enclosures: the first is a formal response to your 
recommendation that C-MANPADS Program Office use a knowledge-based management approach. 
The second attachment contains comments or suggests word or phrase changes that we hope would 
prove useful in clarifying the C-MANPADS SPO's direction. 

Again, thank you for the opportunity to comment. 

Respectfully, 

Dr. Charles E. McQueary/ / 
Under Secretary for Sciwice and Technology 

Enclosures: 
As stated 



24 



GAO-04-341R Missile Protection for Commercial Aircraft 



Enclosure II 



Enclosure II 



GAO DRAFT REPORT - DATED JANUARY 12, 2004 

(GAO-04-341R) 



'DHS NEEDS TO FULLY ADOPT A KNOWLEDGE-BASED APPROACH TO ITS COUNTER- 
MANPADS DEVELOPMENT PROGRAM" 



DHS COMMMENTS ON THE GAO RECOMMENATION 



Now on p. 6. 



RECOMMENDATION : GAO recommends that the Secretary of Homeland Security fully adopt 
the knowledge-based approach, including the use of exit criteria, to help ensure that key decisions in 
DHS' effort to develop and demonstrate a counter-MANPADS system are based on sufficient 
information, (p. 8/GAO draft report) 



DHS RESPONSE : Concur. As indicated in subject report, the DHS Counter-MANPADS Special 
Program Office (SPO) began to incorporate knowledge-based exit criteria into Counter-MANPADS 
solicitation based on input received from GAO during the course of their review. DHS remains 
committed to the adoption of a knowledge-based approach throughout the program and has 
established knowledge-based entrance and exit criteria as an integral part of each milestone. 



(120265) 
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This is a work of the U.S. government and is not subject to copyright protection in the 
United States. It may be reproduced and distributed in its entirety without further 
permission from GAO. However, because this work may contain copyrighted images or 
other material, permission from the copyright holder may be necessary if you wish to 
reproduce this material separately. 
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